Systems and methods for mapping message contents to virtual physical properties for vibrotactile messaging

ABSTRACT

Systems and methods for mapping message contents to virtual physical properties for vibrotactile messaging are disclosed. For example, one disclosed method includes the steps of displaying a first visual representation of a virtual message object having an associated virtual physical parameter at a first device, generating a haptic effect based on the virtual physical parameter at the first device, and sending a message to a second device, the message being configured to display a second visual representation of the virtual message object at the second device and generate the haptic effect based on the virtual physical parameter at the second device.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims priority to: U.S.patent application Ser. No. 15/295,030, filed Oct. 17, 2016, which is acontinuation of, and claims priority to, U.S. patent application Ser.No. 14/978,384, filed Dec. 22, 2015 and which issued as U.S. Pat. No.9,477,309 on Oct. 25, 2016, which is a continuation of and claimspriority to U.S. patent application Ser. No. 14/846,059, filed Sep. 4,2015 and which issued as U.S. Pat. No. 9,229,532 on Jan. 5, 2016, whichis a continuation of and claims priority to: U.S. patent applicationSer. No. 14/491,137, filed Sep. 19, 2014 and which issued as U.S. Pat.No. 9,134,803 on Sep. 15, 2015, which is a continuation of and claimspriority to: U.S. patent Ser. No. 14/069,892, filed Nov. 1, 2013 andwhich issued as U.S. Pat. No. 8,866,602 on Oct. 21, 2014, which is acontinuation of and claims priority to U.S. patent application Ser. No.12/502,647, filed Jul. 14, 2009 and which issued as U.S. Pat. No.8,587,417 on Nov. 19, 2013, which claims priority to U.S. ProvisionalPatent Application No. 61/080,978, filed Jul. 15, 2008; U.S. ProvisionalPatent Application No. 61/080,981, filed Jul. 15, 2008; U.S. ProvisionalPatent Application No. 61/080,985, filed Jul. 15, 2008; U.S. ProvisionalPatent Application No. 61/080,987, filed Jul. 15, 2008; U.S. ProvisionalPatent Application No. 61/148,312, filed Jan. 29, 2009; and U.S.Provisional Patent Application No. 61/181,280, filed May 26, 2009; theentirety of all of which are hereby incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to messaging, and moreparticularly to systems and methods for mapping message contents tovirtual physical properties for vibrotactile messaging.

BACKGROUND

Conventional tactile feedback in messaging systems is determined by thesettings on a message recipient's messaging device, which may associatedifferent types of messages with particular vibrations. These vibrationsmay be generic, and/or based on a prepackaged library of vibrationeffects included with the messaging device. Such vibrations are notauthored by the user, and thus may not be considered personal to thesender or recipient of the message. Furthermore, such vibrations are notgenerated by the properties of the message contents. Accordingly, thereis a need for systems and methods for mapping message contents tovirtual physical properties for vibrotactile messaging.

SUMMARY

Embodiments of the present invention provide systems and methods formapping message contents to virtual physical properties for vibrotactilemessaging. For example, in one embodiment, a method includes displayinga first visual representation of a virtual message object having anassociated virtual physical parameter at a first device, generating ahaptic effect based on the virtual physical parameter at the firstdevice, and sending a message to a second device, the message beingconfigured to display a second visual representation of the virtualmessage object at the second device and generate the haptic effect basedon the virtual physical parameter at the second device.

These illustrative embodiments are mentioned not to limit or define theinvention, but to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description, andfurther description of the invention is provided there. Advantagesoffered by various embodiments of this invention may be furtherunderstood by examining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention are better understood when the following Detailed Descriptionis read with reference to the accompanying drawings, wherein:

FIG. 1 is a block diagram of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 2 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 3 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 4 is a flow diagram of a method for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 5 is a flow diagram of a method for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 6 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention; and

FIG. 7 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide systems and methods formapping message contents to virtual physical properties for vibrotactilemessaging.

Illustrative Embodiment of Vibrotactile Messaging

In one illustrative embodiment of the present invention, a messagingdevice such as a mobile phone comprises a processor in communicationwith a sensor and an actuator. The actuator is configured to generate ahaptic effect. In one such illustrative embodiment, the messaging devicecomprises the Samsung SGH-i710 mobile computer equipped with ImmersionCorporation's VibeTonz® vibrotactile feedback system. In anotherembodiment, the messaging device comprises Immersion CorporationsTouchSense® Technology system also known as Immersion TouchSense®vibrotactile feedback system. Other messaging devices and hapticfeedback systems may be utilized.

The messaging device may be capable of creating messages, represented asvirtual message objects, and exchanging them with other messagingdevices. Virtual message objects in the form of a ball, a tube, or anegg, may represent various types of messages, such as a non-verbalmessage (e.g. a smile), a text message, or a message with a fileattachment. In addition to their shape and appearance, virtual messageobjects may have other properties, or virtual physical parameters, suchas size, mass, collision behavior, and/or texture.

When a new virtual message object is created, a user may define one ormore virtual physical parameters of the object through gestures on thetouchscreen or on the device itself. A processor may receive suchgestures in the form of a sensor signal, and determine a virtualphysical parameter of the new virtual message object based at least inpart on the sensor signal. For example, a user may create a new virtualmessage ball. Next, the user shakes the messaging device to define theelasticity of the new virtual message ball. A sensor such as anaccelerometer or a gyroscope detects the shaking as changes inacceleration, inclination, inertia, or location, and transmits a sensorsignal. The processor receives the sensor signal, and determines avirtual physical parameter of the new virtual message object, such as anelastic surface or behavior, imitating a real rubber ball.

After determining a virtual physical parameter of the new virtualmessage object based at least in part on the sensor signal, theprocessor determines a haptic effect based at least in part on thevirtual physical parameter. Based on a fast jabbing motion, theprocessor may determine that the new virtual message object is brittle,or fragile. A haptic effect based on a brittle or fragile virtualphysical parameter may comprise a vibration imitating an egg cracking orwindow breaking. If the virtual message object is defined as small andelastic, a corresponding haptic effect may imitate a rubber ballbouncing against a wall. The processor may map or translate one, some,or all of the virtual physical properties into haptic effects associatedwith the virtual message object.

Finally, the processor generates a haptic signal configured to cause anactuator to output the haptic effect. The haptic signal is then outputto the actuator which produces the haptic effect. For example, as thevirtual message object is contacted through a touch-screen, themessaging device may vibrate or shake, simulating the effect of touchingor feeling the virtual message object. By generating haptic feedback,the messaging device can convey one or more virtual physical parametersassociated with the virtual message.

This illustrative example is given to introduce the reader to thegeneral subject matter discussed herein. The invention is not limited tothis example. The following sections describe various additionalembodiments and examples of methods and systems for mapping messagecontents to virtual physical properties for vibrotactile messaging.

Mapping Message Contents to Virtual Physical Properties

Embodiments of the invention presented herein provide systems andmethods for mapping message contents to virtual physical properties forvibrotactile messaging. An application for mapping message contents maytake sensor data as input, extract a feature from the sensor data, mapthe feature to a virtual physical parameter, and synthesize a hapticeffect based on the virtual physical parameter. Mapping message contentsto virtual physical properties can facilitate non-linguisticcommunication through audio, visual, and haptic feedback which models aphysical metaphor.

Virtual Message Objects and Virtual Physical Properties

In a physical metaphor, virtual objects can behave like real, physicalobjects. The properties, or virtual physical parameters, of each virtualobject inform how each virtual message object behaves. Thus, in aphysical metaphor of a messaging system, messages such as text messages,voicemail, or file attachments are represented as virtual messageobjects with virtual physical parameters. Such virtual message objectsmay populate a multi-dimensional message inbox, or a virtual messageenvironment. Inside the virtual message environment, virtual messageobjects can interact with each other and behave like real, physicalobjects.

Each virtual message object may represent an individual message, such asa text message, a picture message, a video message, a voicemail, areminder, or a non-verbal message such as a smiley-face or a frown. Forexample, a virtual message object in the form of a tube or rolled-upscroll may represent a text message. And a virtual message object in theform of an egg or a capsule may represent a message with an attachment,such as a video, picture, or music file. Virtual message objects mayalso represent various forms of non-verbal communication, such asgestures, facial expressions, or emotions. A virtual message ball maycorrespond to a non-verbal message such as an indication of playfulness.Or a virtual message arrow may represent a poke or a jabbing gesture(i.e. to request or demand attention). Some effective virtual messageobjects may have easily identifiable visual and/or haptic propertiesand/or effects. These properties help a user intuitively understand thecontents of a message without physically looking at the device. Forexample, the clanging keys of a virtual message object in the form of atypewriter may represent a message from work or a word processing file.

Inside a virtual message environment, virtual message objects caninteract with each other and the environment, such as by bouncing,rolling, or even breaking. The behavior of virtual message objects, orhow they interact, is based at least in part on each virtual messageobject's properties, or virtual physical parameters. Some virtualphysical parameters, such as size, mass, shape, collision behavior,texture, or visual representation, are relatively static parameters, orinherent properties of the virtual message object itself. Other virtualphysical parameters, such as urgency, lifetime, and/or attitude, may betransitory or temporary parameters. Such variable virtual physicalparameters may be associated with how the virtual message object is sentor received from one device to another. The virtual physical parametersof a virtual message object may be based at least in part on propertiesof the message itself. For instance, the file size of a messageattachment, a length of a text message, an area or a resolution of apicture attachment, a length of a song attachment, or a length of avideo attachment may be used to determine the size or mass of a virtualmessage object.

In some embodiments, virtual physical parameters are determined based onsensor signals generated when a user interacts with the messagingdevice. For instance, a user may move or interact with a device tocreate and/or define a new virtual message object. In one embodiment, auser blows on a microphone to inflate a new virtual message balloon. Thevirtual message balloon's virtual physical size may directly correlateto the length of time the user blew on the microphone. In anotherembodiment, a user with a picture phone may define a new virtual messageobject by taking a picture. For example, after taking a picture of abasketball, the messaging device is assigned properties, such as size,texture, or appearance, extracted from or based at least in part onproperties of the picture.

The virtual message environment may also implement a token metaphor forits contents. In a token metaphor, virtual objects are passed back andforth between devices, mimicking the exchange of a definite physicalobject. A token, such as a virtual message object, may be passed fromone messaging device to another. When one messaging device sends avirtual message object to another messaging device, the virtual messageobject disappears from the sending messaging device, as if it wasphysically passed to the receiving messaging device.

When a messaging device receives a new virtual message object, themessaging device may output effects, such as haptic effects, to signifythat a new virtual message object has been received, and is in thevirtual message environment. For example, when a virtual message ball isreceived, sound and haptic effects may be played to mimic a bouncingball, signifying to the user that a new virtual message ball has beenreceived.

The audio, visual, and/or haptic effects associated with virtual messageobjects may be determined based at least in part on one or more virtualmessage properties of a virtual message object. A thin, egg-shelledvirtual message object may crack or break upon impact with the virtualmessage environment. Alternatively, a toy ball or ping-pong ball maycontinually bounce off the boundaries of the virtual messageenvironment. A virtual message object in the form of a bubble may beassociated with a brief, pointed haptic effect, indicative of a bubblebursting.

Illustrative Systems for Mapping Message Contents to Virtual PhysicalProperties

Referring now to the drawings in which like numerals indicate likeelements throughout the several Figures, FIG. 1 is a block diagram of asystem for mapping message contents to virtual physical properties forvibrotactile messaging according to one embodiment of the presentinvention. As shown in FIG. 1, the system 100 comprises a messagingdevice 102, such as a mobile phone, portable digital assistant (PDA),portable media player, or portable gaming device. The messaging device102 comprises a processor 110. The processor 110 is in communicationwith a network connection 112, a sensor 114, a display 116, an actuator118, and a speaker 120. The messaging device 102 may show a virtualmessage environment generated by the processor 110 on the display 116.

The processor 110 is in communication with the network connection 112.The network connection 112 may comprise one or more methods of mobilecommunication, such as infrared, radio, Wi-Fi, or cellular networkcommunication. In other variations, the network connection 112 comprisesa wired network connection, such as an Ethernet connection or a modem.The messaging device 102 can be configured to exchange messages, such asvoice messages, text messages, data messages, or virtual message objectswith other messaging devices (not shown) over networks such as acellular network or the Internet.

The processor 110 is also in communication with one or more sensors 114.The sensor 114 may comprise an accelerometer, a gyroscope, a GPS sensor,a touch-sensitive input device (e.g. touch screen, touch-pad), a texturestylus, an imaging sensor, or some other type of sensor. The one or moresensors 114 may be configured to detect changes in acceleration,inclination, inertia, or location. For example, the messaging device 102may comprise an accelerometer configured to measure acceleration of themessaging device 102. Or the messaging device 102 may comprise alocation sensor, rotary velocity sensor, light sensor, pressure sensor,texture sensor, camera, microphone, or other type of sensor. In theembodiment shown, the one or more sensors 114 are configured to send asensor signal to the processor 110. The processor 110 can be configuredto receive the sensor signal from the one or more sensors 114.

In some embodiments of the present invention, users record virtualphysical parameters of a virtual message object through interactions,such as movements or gestures. These physical interactions are detectedby the one or more sensors 114. As the messaging device 102 is tilted,shaken, or otherwise moved, the one or more sensors 114 may detect thesemovements, and generate a sensor signal based at least in part on themovement of the messaging device 102. In one embodiment, anaccelerometer sensor is configured to detect the inclination andacceleration of the messaging device 102. As the messaging device 102 istilted, the accelerometer can be configured to send signals to theprocessor based at least in part on the tilt and/or acceleration of themessaging device 102. In another embodiment, the display 116 comprises atouch-screen configured to detect gestures or position inputs. As afinger is positioned or dragged on the touch-screen display 116, thetouch-screen is configured to send signal to the processor 110 based atleast in part on the finger's movement.

Upon receiving a sensor signal, the processor 110 may be configured todetermine a virtual physical parameter of a virtual message object basedat least in part on the sensor signal. Virtual physical parameters maybe determined based on features extracted from sensor signals, such ascolor, texture, or speed. After determining a virtual physicalparameter, the processor 110 may be configured to determine a hapticeffect based at least in part on the virtual physical parameter. Ahaptic effect may simulate a rough-textured virtual message throughvibrations or hard, irregular jolts.

In one embodiment, a user selects a new virtual message egg from a menu.Next, the user may move the messaging device 102 in a large circle oroval, to indicate the size of the virtual message egg. After determiningthe large size of the virtual message egg based on the large movement ofthe messaging device 102, the processor 110 may determine a hapticeffect simulating the cracking of a large egg, such as through a numberof vibrations.

As shown in FIG. 1, the processor 110 is in communication with a display116. The processor 110 can be configured to generate a graphicalrepresentation of a virtual message environment, to be shown on display116. The display 116 may comprise a touch-sensitive input device, suchas a touch screen, configured to send and receive signals from theprocessor 110. Virtual message objects shown on the display 116 may bedirectly manipulated through a touch-screen. For example, twodimensional finger gestures on a touch screen display may select, touch,feel, drag, or throw a virtual message object within the virtual messageenvironment.

The processor 110 is also in communication with one or more actuators118. The processor 110 may transmit a haptic signal associated with ahaptic effect to the one or more actuators 118. The actuators 118receive haptic signals associated with haptic effects, and then outputhaptic effects. The actuator 118 may be, for example, an electric motor,an electro-magnetic actuator, a voice coil, a linear resonant actuator,a piezoelectric actuator, a shape memory alloy, an electro-activepolymer, a solenoid, an eccentric rotating mass motor (ERM) or a linearresonant actuator (LRA).

Finally, in the embodiment shown, the processor 110 is in communicationwith a speaker 120. The processor 110 may be configured to determine asound effect based at least in part on the interaction between a virtualmessage object and the virtual message environment, and send an audiosignal to the speaker based at least in part on the sound effect. Thespeaker 120 can be configured to generate the sound effect based atleast in part on the audio signal generated by the processor 110. Theoutput of a sound effect and a haptic effect may be coordinated, orgenerated at the same time as the haptic effect. In one embodiment, thespeaker may generate the sound effect substantially simultaneously asthe actuator generates the haptic effect.

Sound effects generated by the processor 110 may simulate an interactionbetween a virtual message object and the virtual message environment,and/or an interaction between a first virtual message object and one ormore other virtual message objects. Examples of sound effects include aball bouncing against a wall, an arrow hitting a wall with a thud, or anegg cracking. For example, a messaging device 102 may receive a virtualmessage object in the form of a water balloon from another messagingdevice. Upon the virtual message object's arrival, a sound effect of awater balloon splashing against a wall may be generated at the same timeas a vibration simulating the splash.

FIG. 2 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. The elements of the system 200 aredescribed with reference to the system depicted in FIG. 1. A variety ofother implementations are possible.

As shown in FIG. 2, the system 200 comprises a messaging device 102,such as a mobile phone, portable digital assistant (PDA), portable mediaplayer, or portable gaming device. The messaging device 102 can beconfigured to send and receive signals, such as voice mail, textmessages, and other data messages, over a network such as a cellularphone network or the Internet. The messaging device 102 may include awireless and/or a wired network connection 112. Although the device 102is illustrated as a handheld messaging device in FIG. 2, otherembodiments may use other devices, such as video game systems, videogame controllers, personal media players, personal digital assistants,and/or personal computers to send and receive virtual message objects.

As shown in FIG. 2, the messaging device 102 comprises a display 116,buttons 122 a, 122 b, 122 c, 122 d, and a track ball 124. In addition tothe buttons 122 a, 122 b, 122 c, 122 d, and track ball 124, themessaging device 102 may comprise a directional pad, a touchpad, ascroll wheel, a rocker switch, a joystick, or other forms of input (notshown in FIG. 2).

The display 116 of the mobile device 102 may comprise a touch-sensitiveinput device, such as a touch-screen. The display 116 can be configuredto receive signals from the processor 110, and generate a graphicalenvironment, such as an inbox, a browser, a gaming environment, avirtual message environment. The display 116 may also be configured togenerate a virtual message authoring environment 250. The virtualmessage authoring environment 250 may comprise a menu 252 for selectinga type of virtual message object to author or create. As shown in FIG.2, the menu 252 comprises options for authoring a new virtual messageball 254 a, a new virtual message tube 254 b, or a new virtual messageegg 254 c. The menu 252 may comprise other types of virtual messageobjects such as a heart or a water balloon (not shown in FIG. 2).

The virtual message authoring environment 250 may comprise an authoringindicator 256. The authoring indicator 256 may alert a user that themessaging device 102 is in authoring or recording mode. As shown in FIG.2, the authoring indicator 256 comprises a warning bar located above themenu 252, at the top of the virtual message authoring environment 250.The virtual message authoring environment may be configured to displaythe authoring indicator 256 when the sensor is recording or capturinginformation associated with the virtual message object. In one instance,when the record button 122 c is pressed or activated, a recording modeis entered, and the authoring indicator 256 appears. When the recordingmode is stopped, the authoring indicator 256 may dim or disappear.

When the messaging device 102 is in recording mode, one or more sensors114 may record or capture information, such as a gesture, associatedwith a virtual physical parameter. Some gestures comprisetwo-dimensional gestures received by a touch-screen. Examples oftwo-dimensional gestures include pointing (i.e. positional input) ordrawing. Other gestures may comprise three-dimensional gestures measuredby an accelerometer, a gyroscope, or some other sensor.Three-dimensional gestures may include rotating, flicking, jabbing, orotherwise moving the entire messaging device 102. One or more sensors114 may also record other non-verbal information, such as by sensingpressure on a pressure sensor, taking a picture with a camera, orrecording a sound with a microphone.

The virtual message authoring environment 250 may also comprise a newvirtual message object pipe 258. As one or more new virtual messageobjects are created, they may appear as if they are coming out of thenew virtual message object pipe 258 and into the virtual messageauthoring environment 250. The new virtual message object pipe 258 mayreinforce the token metaphor of a virtual message object. For example,when a new virtual message object is created, and emerges from the newvirtual message object pipe 258, a user of the messaging device 102 mayintuitively deduce that the new virtual message object represents aphysical token that can be passed back and forth between differentmessaging devices.

FIG. 3 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. As shown in FIG. 3, the system 300comprises a messaging device 302 with a touch-screen display 316,texture sensor 314, and track ball 324.

The touch-screen display 316 is showing a virtual message authoringenvironment 350. As shown in FIG. 3, the virtual message authoringenvironment 350 comprises a menu 352 for selecting the type of a newvirtual message object. In menu 352 a user may select a new virtualmessage ball 354 a, a new virtual message tube 354 b, or a new virtualmessage egg 354 c. Although the menu 352 illustrates three types ofvirtual message objects, other types of virtual message objects may beselected (not shown in FIG. 3). The virtual message authoringenvironment 350 also comprises a new virtual message pipe 358. When anew virtual message ball 354 a is selected, a new virtual message ball360 may be shown exiting the new virtual message pipe 358.

During creation of a new virtual message object, such as new virtualmessage ball 360, the virtual message authoring environment 350 maydisplay a slider bar 362. A user may adjust the slider bar 362 to changeone or more virtual physical properties of a new virtual message object.In FIG. 3, the slider bar 362 is being used to adjust the texture of thevirtual message object 360. On one end of the slider bar, an icon 364 ashown as a stack of rocks illustrates one texture that can be assignedto the new virtual message object 360. In the middle of the slider bar362, a second icon 364 b shown as a pile of course sand illustrates asecond texture that can be assigned to the virtual message object 360.On the other end of the slider bar, an icon 364 c shown as a pile offine sand illustrates a third texture that can be assigned to thevirtual message object 360.

The slider bar 362 may be manipulated or engaged through thetouch-screen 316, through the track ball 324, or through some othermeans. Alternatively, one of the icons 364 a, 364 b, 364 c can beengaged through the touch-screen 316. As the slider bar 362 is engaged,the touch-sensitive sensor sends a signal to a processor of themessaging device 302. The processor receives the sensor signal, and candetermine a virtual physical property of the virtual message objectbased at least in part on the sensor signal associated with the sliderbar input.

Illustrative Methods for Mapping Message Contents to Virtual PhysicalProperties for Vibrotactile Messaging

FIG. 4 is a flow diagram of a method for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. In the method 400, a processorreceives a sensor signal from a sensor 402. The processor may be incommunication with one or more sensors, such as an accelerometer, agyroscope, a GPS sensor, a touch-sensitive input device (e.g. touchscreen, touch-pad), a texture stylus, a pressure sensor, an imagingsensor, a microphone, or some other type of sensor. A user may create anew virtual message object, such as a water balloon, by selecting awater balloon type from a menu, entering a record mode, and tipping themessaging device downward to fill the water balloon. An accelerometer ora gyroscope may detect the change in inclination, a touch-screen maydetect a figure drawn on its surface, or a microphone may detect sound.After detecting such gestures or actions, a sensor signal may betransmitted to the processor of the messaging device.

Next the processor determines a virtual physical parameter of a virtualmessage object based at least in part on the sensor signal 404. Afterreceiving a sensor signal from a sensor, the processor may extractgestural and/or environmental features from the sensor signal. Agestural feature may comprise information related to the movement of thedevice, or a movement on the device. An environmental feature maycomprise information related to the surroundings or condition of thedevice. Gestural and/or environmental extraction algorithms may beapplied to data received from a sensor. Environmental features, such asthe shape or color of an object in a picture, may be extracted from thesensor signal. Alternatively, the direction the messaging device wasmoved, or a character input on a touch-screen may be extracted from thesensor signal. A threshold may be set such that a certain acceleration,orientation, or periodicity triggers the creation of a token. Forinstance, if the device is only moved a small distance, the processormay determine that the movement was inadvertent, or not meant to triggerthe creation of a token. If the device is moved a large distance, theprocessor may determine that the movement was intended to trigger thecreation of a token.

The processor may determine a virtual physical parameter of the newvirtual message object by mapping gestural and/or environmental featuresextracted from the sensor signal to a virtual physical parameter. Forexample, a predetermined set of gestural and/or environmental featuresmay be stored in a data store along with a corresponding virtualphysical parameter. When the gestural and/or environmental feature isextracted, the data store is searched, and the appropriate virtualphysical parameter is determined. The determination may be based atleast in part on the dynamics of a gesture, such as the speed of thetouch-screen contact, speed of the movement, size of the movement,periodicity of movement, or repetition of the movement.

As one example, a user may create a plurality of new virtual messageobjects, such as hearts or balloons through a gentle, repetitive motion.With each cycle, or repetitive gesture, the processor may create a newvirtual message object, potentially leading to a plurality of balloonsfilling up the virtual message authoring environment. In a furtherexample, a user may engage or touch each balloon through thetouch-screen display. By slowly dragging the balloon up or down on thetouch-screen, the processor may determine a virtual physical weight ofthe balloon. If the balloon is slowly dragged upwards, then theprocessor may determine the balloon is light, and should float around inthe virtual message environment. Alternatively, if the balloon isquickly dragged to the bottom of the display, the processor maydetermine that the balloon is dense and/or heavy, and should sink to thebottom of the virtual message environment.

The virtual physical parameters of virtual message objects may be basedat least in part on other factors. A file, such as a video file or animage file, can be associated with a virtual message object, such as avirtual message capsule. The processor may extract metadata from thefile, use that metadata to determine a virtual physical parameter. Inone example, a user creates a new virtual message capsule. Next, theuser scrolls through a list of files, and selects a particular file toattach to the virtual message capsule by dragging it onto the virtualmessage capsule. The processor may determine the size of the virtualmessage capsule based at least in part on the size of the particularfile dragged onto the virtual message object. As another example, theprocessor may determine the texture or color of a virtual messagecapsule based on the type of file that is associated with the capsule.

After determining a virtual physical parameter, the processor determinesa haptic effect based at least in part on the virtual physical parameter406. For example, a user may create a new virtual message ball andjostle the device to define a bouncy collision behavior for the newvirtual message ball. The processor may determine a haptic effectsimulating the virtual message ball bouncing against the virtual messageenvironment, mimicking the gesture used to define the virtual messageball when it was created.

If the virtual message environment contains a single virtual messageobject such as a large, brittle egg, the processor may determine ahaptic effect simulating the egg breaking (e.g. a single large crack).As another example, if there are many virtual message objects comprisingkinetic bouncy balls, the processor may determine a large number ofsmall, quick vibrations are appropriate, to simulate the kinetic ballsbouncing off the environment and each other. By associating the hapticeffects with the virtual physical parameters of the virtual messageobjects, a user may correlate certain haptic effects with certain typesof objects. For example, when the actuator outputs a haptic effectsimulating a beating heart, the user may immediately guess that she hasreceived an intimate message, or virtual message object, without lookingat the messaging device.

The processor generates a haptic signal configured to cause an actuatorto output the haptic effect 408. Finally, the processor transmits thehaptic signal to one or more actuators configured to output the hapticeffect 410. In one variation, the processor determines a haptic effectsimulating an arrow hitting a wall, such as a thud, and generates ahaptic signal configured to cause an actuator to output the thud. Thehaptic signal is then transmitted to the actuator, which outputs thehaptic effect.

FIG. 5 is a flow diagram of a method for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. In the method 500, a new virtualmessage object is created 502. New virtual message objects may becreated in a virtual message authoring environment. The virtual messageauthoring environment may suspend the physical model of themulti-dimensional inbox, or virtual message environment. By suspendingthe physical model, sensor signals may be associated with a virtualphysical parameter of a new virtual message object, rather thanassociating a sensor signal with an interaction with an existing virtualmessage object inside the virtual message environment.

A new virtual message object may be created by selecting an option froma menu. The virtual message authoring environment may display ascrolling menu for choosing one of several types of existing virtualmessage objects, and/or an option for creating a new type of virtualmessage object. As shown in FIG. 5, a new virtual message ball 504 a, anew virtual message tube 504 b, or a new virtual message capsule 504 ccan be created. Other types of virtual message objects may be created inother embodiments. For example, a new virtual message object maycomprise a heart, an arrow, a water balloon, or a basketball. Each typeof virtual message object may have specific virtual message propertiesassociated with it. For example, a virtual message basketball may have avirtual physical texture simulating a real basketball.

In some instances, a user selects the virtual message object from a listof pre-defined virtual message object types. By using a predefinedvirtual message object type, a user may be able to quickly andexplicitly define specific aspects of the new virtual message object,which can be mapped or translated into one or more virtual physicalparameters. In other instances, a user may author a new type of virtualmessage object from scratch. The user may be able to define a virtualmessage object with specific, individualized virtual physical parametersbased at least in part on signals from a touch-screen, an accelerometer,a gyroscope, GPS sensor, or some other type of sensor.

After a new virtual message ball is created 504 a, a feature of thevirtual message ball is recorded 506 a. Features may be recorded by oneor more sensors of a messaging device. A color feature may be recordedfrom a photo sensor, a texture feature may be recorded from a texturestylus sensor, or a gesture may be recorded from an accelerometer.

In step 504 b, a new virtual message tube is created. A virtual messageobject in the form of a tube, or a scroll, may represent a text message.Thus, if a new virtual message tube is created 504 b, then a textmessage may be entered 506 b. A user may enter text through a keypadattached to the messaging device, through a virtual keyboard shown on atouch-screen display, or through some other means.

In step 504 c, a virtual message capsule is created. A virtual messageobject in the form of a capsule or an egg may represent a message with afile attachment. Thus, if a new virtual message capsule is created 504c, then a file to attach (or include inside the capsule) is selected 506c. A virtual message capsule may comprise a message with a fileattachment, such as a video file, a sound file, or an image file. A usermay select a file attachment by scrolling through a list of files, ordragging a file onto a virtual message capsule. As one alternative, auser may take a picture and select the picture to enclose within thefile capsule.

After information about the new virtual message object is recorded 506a, entered 506 b, or selected 506 c, a virtual physical parameter of thenew virtual message object is determined 508. The virtual physicalparameter may be determined based at least in part on informationreceived from a sensor. When a sensor records a feature of the virtualmessage ball 506 a, the virtual physical parameter may be determinedbased at least in part on information recorded by the sensor. Forexample, a user may tilt or make a gesture with the messaging device torecord a feature associated with the size of the new virtual messageobject. The virtual physical size of the new virtual message object maythen correspond to the size of the gesture or movement, a small movementof the messaging device may produce a small virtual message object,while a large movement or gesture of the messaging device may produce alarge virtual message object.

When a new virtual message tube is created 506 b, the length of the textmessage (such as the number of lines or the number of words) maydetermine, at least in part, one or more virtual physical parameters ofthe new virtual message tube, such as the tube's size or length. Ahaptic effect based on the size or length of the virtual message tubemay be determined. Thus, haptic sensations may convey information aboutthe virtual message tube.

In the case of a new virtual message capsule 504 c, the size of the fileselected 506 c may determine the virtual mass of the new virtual messageobject. Because a haptic effect may be based at least in part on thevirtual mass of the virtual message object, a haptic effect particularlydesigned for that virtual message object can be generated. The hapticsensations that result may convey information about the file capsulesuch as the amount of data it contains or the type of data.

A haptic signal is generated to cause an actuator to output a hapticeffect based at least in part on the virtual physical parameter of thenew virtual message object 510. Finally, the haptic signal istransmitted to an actuator configured to output the haptic effect 512.The haptic effect may be based at least in part on an interactionbetween the virtual message object and the virtual message environment.The behavior of the virtual message object may be based at least in parton the virtual physical parameters of the virtual message object. Anegg-shelled virtual message object may crack or break upon impact withthe virtual message environment. Alternatively, a virtual message arrowmay arrive with one resounding thud. A virtual message bubble may beassociated with a brief, pointed haptic effect, indicative of a bubblebursting.

If the new virtual message object is a large ball, then a haptic signalmay be generated to cause an actuator to output a haptic effectmimicking the large ball bouncing and/or rolling around in the virtualmessage environment. If the new virtual message object is a smallcapsule, then a haptic signal may be generated to cause an actuator tooutput a haptic effect mimicking the small capsule breaking inside thevirtual message environment.

Haptic effects may be associated with virtual message objects. Thehaptic effects may be preserved as the virtual message object isexchanged between messaging devices. For example, a processor mayassociate a virtual message capsule with specific haptic effects. Whenthe capsule is sent from one mobile device to another, the haptic datais also sent to the receiving device. This haptic data may be used togenerate haptic feedback in the receiving device.

FIG. 6 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. As shown in FIG. 6, the system 600comprises a messaging device 602. The messaging device 602 displays avirtual message authoring environment 650. The messaging device 602includes one or more processor(s) (not shown in FIG. 6).

Additionally, the messaging device 602 may include one or more sensors(not shown in FIG. 6). The sensor, such as an accelerometer, may beconfigured to detect changes in acceleration, inclination, inertia, orlocation of the messaging device. As other examples, the messagingdevice may include an air pressure sensor, location sensor, rotationalvelocity sensor, image sensor, pressure sensor, or some other type ofsensor. The sensor(s) are configured to send a sensor signal to aprocessor.

In one variation, acceleration sensing can be used to capture a gestureor movement associated with creating a virtual message object. Inanother variation, a touch-sensitive input device such as a touch-screenis used to capture a drawing associated with a new virtual messageobject. A gyroscopic sensor may detect changes in the orientation of themessaging device in the x and y planes. As shown in FIG. 6, the mobiledevice 602 is moved in a direction according to arrow 610. The processormay determine that a virtual message object created from the gestureillustrated in FIG. 6 is large, round, and readily rolls around avirtual message environment. A corresponding haptic effect may be agentle, smooth vibration.

FIG. 7 is an illustration of a system for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention. As shown in FIG. 7, the system 700comprises a messaging device 702. The messaging device displays avirtual message authoring environment 750. The messaging device 702 mayinclude one or more processors (not shown in FIG. 7) and one or moresensors (not shown in FIG. 7).

As shown in FIG. 7, a user moves or shakes the messaging device inshort, horizontal motions indicated by the direction of arrow 710. Thesensor data extracted from each movement or gesture illustrated in FIG.7 can be used to create a new virtual message object with specificvirtual physical properties. The processor may determine that a virtualmessage object created from the gesture illustrated in FIG. 7 may be asmall, rough object. A corresponding haptic effect may be a successionof quick jolts.

Each haptic effect may be associated with the virtual message object andpreserved. For example, the haptic effect may be included in the profileof the virtual message object. In this way a messaging device whichreceives the virtual message object also receives the signalcorresponding to haptic effect associated with that virtual messageobject. In such a manner, users may communicate through non-verbalhaptic messages.

Embodiments of the present invention can be implemented in digitalelectronic circuitry, or in computer hardware, firmware, software, or incombinations of them. In one embodiment, a computer may comprise aprocessor or processors. The processor comprises a computer-readablemedium, such as a random access memory (RAM) coupled to the processor.The processor executes computer-executable program instructions storedin memory, such as executing one or more computer programs formessaging. Such processors may comprise a microprocessor, a digitalsignal processor (DSP), an application-specific integrated circuit(ASIC), field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage or transmissiondevice capable of providing a processor, such as the processor in a webserver, with computer-readable instructions. Other examples of mediacomprise, but are not limited to, a floppy disk, CD-ROM, magnetic disk,memory chip, ROM, RAM, ASIC, configured processor, all optical media,all magnetic tape or other magnetic media, or any other medium fromwhich a computer processor can read. Also, various other devices mayinclude computer-readable media, such as a router, private or publicnetwork, or other transmission device. The processor, and theprocessing, described may be in one or more structures, and may bedispersed through one or more structures. The processor may comprisecode for carrying out one or more of the methods (or parts of methods)described herein.

GENERAL

The foregoing description of the embodiments, including preferredembodiments, of the invention has been presented only for the purpose ofillustration and description and is not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Numerousmodifications and adaptations thereof will be apparent to those skilledin the art without departing from the spirit and scope of the invention.

What is claimed is:
 1. A method comprising: generating, at a messagingdevice, a virtual message object based on one or more user inputs;determining and mapping, by the messaging device, a virtual physicalparameter to the virtual message object based on one or more propertiesof the virtual message object; determining, by the messaging device, abehavior of a virtual message object in a virtual message environmentbased on the virtual physical parameter and a physical interaction witha messaging device; determining and imparting, by the messaging device,a haptic effect to the virtual message object based on the determinedbehavior; and transmitting, by the messaging device, the virtual messageobject to a remote device based on the behavior, the virtual messageobject comprising the virtual physical parameter and the haptic effect.2. The method of claim 1, wherein determining the virtual physicalparameter is further based on a user interaction with the messagingdevice.
 3. The method of claim 2, wherein the user interaction comprisesa gesture, and wherein determining the behavior comprises determining acollision behavior of the virtual message object based on the gesture.4. The method of claim 3, wherein the gesture comprises dragging orthrowing the virtual message object or flicking, jabbing, or moving themessaging device.
 5. The method of claim 1, wherein the virtual physicalparameter comprises a size or a mass, and wherein the size or the massis based on a data size of the virtual message object.
 6. The method ofclaim 5, wherein the data size is a file size of a message attachment, alength of a text message, an area or resolution of a picture attachment,a length of a song attachment, or a length of a video attachment.
 7. Amessaging device comprising: a haptic output device; a display; anon-transitory computer-readable medium; and a processor incommunication with the non-transitory computer-readable medium, theprocessor configured to execute processor-executable instructions storedin the non-transitory computer-readable medium to: generate a firstvirtual message object based on one or more user inputs; determine andmap a first virtual physical parameter to the first virtual messageobject based on one or more properties of the first virtual messageobject; determine a first behavior of the first virtual message objectin a virtual message environment based on the first virtual physicalparameter and a physical interaction with the messaging device;determine and imparting a first haptic effect to the first virtualmessage object based on the determined first behavior; transmit thefirst virtual message object to a remote device based on the firstbehavior, the first virtual message object comprising the first virtualphysical parameter and the first haptic effect; receive a second virtualmessage object from a second remote device, the second virtual messageobject comprising a second virtual physical parameter and a secondhaptic effect; determine a second behavior of the second virtual messageobject based on the second virtual physical parameter and a virtualmessage environment; display, using the display, the second behavior ofthe second virtual message object; and output, using the haptic outputdevice, the second haptic effect corresponding to the displayed secondbehavior of the second virtual message object.
 8. The messaging deviceof claim 7, wherein the second remote device is the remote device. 9.The messaging device of claim 7, wherein determining the first virtualphysical parameter is further based on a user interaction with themessaging device.
 10. The messaging device of claim 9, wherein the userinteraction comprises a gesture, and wherein determining the firstbehavior comprises determining a collision behavior of the first virtualmessage object based on the gesture.
 11. The messaging device of claim10, wherein the gesture comprises dragging or throwing the first virtualmessage object or flicking, jabbing, or moving the messaging device. 12.The messaging device of claim 7, wherein the first or second virtualphysical parameter comprises a size or a mass, and wherein the size orthe mass is based on a data size of the respective first or secondvirtual message object.
 13. The messaging device of claim 12, whereinthe data size is a file size of an attachment to the first or secondvirtual message object, a length of a text message of the first orsecond virtual message object, an area or resolution of a pictureattachment to the first or second virtual message object to the first orsecond virtual message object, a length of a song attachment to thefirst or second virtual message object, or a length of a videoattachment to the first or second virtual message object.
 14. A methodcomprising: receiving, by a messaging device, a virtual message objectfrom a remote device, the virtual message object comprising a virtualphysical parameter and a haptic effect; determining a behavior of thevirtual message object based on the virtual physical parameter and avirtual message environment; displaying, by a display of the messagingdevice, the behavior of virtual message object; and outputting, using ahaptic output device of the messaging device, the haptic effectcorresponding to the displayed behavior of the virtual message object.15. The method of claim 14, wherein the virtual physical parametercomprises a size or a mass, and wherein the size or the mass is based ona data size of the virtual message object.
 16. The method of claim 15,wherein the data size is a file size of a message attachment, a lengthof a text message, an area or resolution of a picture attachment, alength of a song attachment, or a length of a video attachment.
 17. Themethod of claim 14, wherein displaying the behavior comprises displayingthe virtual message object bouncing or rolling within the virtualmessage environment, and wherein outputting the haptic effectcorresponds to a bounce or rolling.
 18. The method of claim 14, whereindisplaying the behavior comprises displaying the virtual message objectcolliding with another virtual message object within the virtual messageenvironment, and wherein outputting the haptic effect corresponds to thecolliding.
 19. The method of claim 14, wherein displaying the behaviorcomprises displaying the virtual message object breaking within thevirtual message environment, and wherein outputting the haptic effectcorresponds to the breaking.
 20. The method of claim 14, furthercomprising: determining a sound effect based on an interaction of thevirtual message object with the virtual message environment based on thebehavior; and outputting, using a speaker of the messaging device, thesound effect corresponding to the interaction.